Text to Point Cloud Localization with Relation-Enhanced Transformer

Automatically localizing a position based on few natural language instructions is essential for future robots to communicate and collaborate with humans. To approach this goal, we focus text-to-point-cloud cross-modal localization problem. Given textual query, it aims identify the described location from city-scale point clouds. The task involves two challenges. 1) In clouds, similar ambient instances may exist in several locations. Searching each huge cloud only as guidance lead less discriminative signals incorrect results. 2) descriptions, hints are provided separately. case, relations among those not explicitly described, leaving difficulties of learning agent itself. alleviate challenges, propose unified Relation-Enhanced Transformer (RET) improve representation discriminability both nature queries. core proposed RET novel Relation-enhanced Self-Attention (RSA) mechanism, which encodes instance (hint)-wise modalities. Moreover, fine-grained matching method further refine predictions subsequent instance-hint stage. Experimental results KITTI360Pose dataset demonstrate that our surpasses previous state-of-the-art by large margins.